1. (Currently Amended) A system for synchronizing multi-modal interactions, 
comprising: 

a program storage device that stores a multi-modal applicatio n, the multi-modal 
application comprising at least a first mode process that enables user interaction with the 
application in a first modality and a second mode process that enables user interaction with the 
application in a second modality; 

a program execution system that executes the multi-modal application and synchronizes 
the first and second mode processes while a user interacts with the multi-modal application, 
wherein the program execution system comprises: 

a multi-modal shell that manages for managing and synchronizing information 
exchanges between the first and second mode processes of the multi-modal application to enable 
a synchronized multi-modal interaction with the multi-modal application wherein user 
interaction in one modality results in execution of corresponding commands in both the first and 
second mode processes : and 

an API (application program interface) that allows the first and second mode processes to 
register their respective active commands and corresponding actions with the multi-modal shell 

2. (Currently Amended) The system of claim 1, wherein the program execution system 
comprises further compri s ing a registry having a registration table, managed by the multi-modal 
shell, wherein the registration table t hat comprises a list of each of the registered commands and 
corresponding synchronized actions that results in both the first and second mode processes upon 
execution of a registered command by one of the first and second mode processes. 

3. (Previously Presented) The system of claim 1, wherein the multi-modal application 
comprises at least a first mono-mode application for the first mode process and a second mono- 
mode application for the second mode process, wherein the multi-modal shell manages and 
synchronizes information exchanges between the first and second mono-mode applications. 

4. (Original) The system of claim 1, further comprising at least a first device having a 
first user interface modality and a second device having a second user interface modality, 
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wherein the multi-modal shell manages and synchronizes information exchanges between the 
first and second devices. 

5. (Original) The system of claim 4, wherein the first device, second device and multi- 
modal shell are distributed over a network, and wherein the API is implemented using distributed 
APIs or protocols. 

6. (Original) The system of claim 2, wherein the API comprises a mechanism for 
converting a mono-mode application to a multi- modal application. 

7. (Original) The system of claim 6, wherein the mono-mode application is a GUI 
application, and wherein the mechanism provides speech enablement of the GUI application by 
registering the active commands of the GUI application and building a grammar for the 
registered commands to support the commands in a speech modality. 

8. (Original) The system of claim 2, wherein the API comprises a mechanism for building 
a multi-modal application. 

9. (Original) The system of claim 8, wherein the mechanism is used for directly 
programming the registry by building a registration table having user-defined commands and 
corresponding actions for each of the modalities of the multi-modal application. 

10. (Original) The system of claim 1, further comprising an operating system, wherein 
the multi-modal shell executes on top of the operating system. 

11. (Original) The system of claim 1, wherein the system is distributed over a network. 

12. (Previously Presented) The system of claim 1, wherein the multi-modal application is 
a multi-modal browser comprising a first browser application and a second browser application. 
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13. (Previously Presented) The system of claim 12, wherein the first browser is a GUI 
browser and the second browser is a speech browser. 

14. (Currently Amended) The system of claim 12, wherein the multi-modal shell 
processes the multi-modal application a CML (conv e rsational markup languag e ) file to send 
modality-specific presentation information in the CML file to the respective browsers. 

15. (Currently Amended) The system of claim 14, wherein the multi-modal application 
is authored using a modality-independent description and wherein the multi-modal shell 
generates CML fil e encapsulates t he modality-specific presentation information in a single from 
the modality-independent description r e pr e s e ntation . 

16. (Currently Amended) The system of claim 14, wherein the multi-modal application 
CML fil e comprises a combination of declarative markup languages to describe the first and 
second mode processes . 

17. (Currently Amended) The system of claim 16, wherein the multi-modal application 
CML file comprise s a singl e fil e combining combines the declarative markup languages and 
synchronization elements in single pages to provide tight synchronization between the first and 
second mode processes d e clarativ e markup languages . 

18. (Currently Amended) The system of claim 16, wherein the multi-modal application 
CML fil e comprises a separate files for each of the declarative markup languages for the first and 
second mode processes and wherein the separate files are loosely synchronized at predefined 
points. 

19. (Currently Amended) A method for synchronizing multi-modal interactions, 
comprising the steps of: 

activating a multi-modal application comprising at least a first mode process that enables 
user interaction with the application in a first modality and a second mode process that enables 
user interaction with the application in a second modality 
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receiving a command or event in the first modality; 

triggering (i) an action in by the first modality mode process based on the received 
command or event and (h) triggering a corresponding action in by the second mode process 
modality, bas e d on th e received command or event ; and 

updating application states or device states associated with the first and second mode 
processes modality and the second modality 

20. (Currently Amended) The method of claim 19, further comprising the steps of: 
registering active commands associated with the first mode process modality and active 

commands associated with the second mode process modality ; 

associating, with each registered command of the first mode process modality , an action 
on the first mode process modality and a corresponding action on the second mode process 
modality ; and 

associating, with each registered command of the second mode process modality , an 
action on the second mode process modality and a corresponding action on the first mode 
process modality . 

21. (Original) The method of claim 20, further comprising the step of building a 
command-to-action registration table based on the registered commands and actions. 

22. (Currently Amended) The method of claim 21, wherein the registration table is built 
by a multi-modal shell via API calls from the applications or devices associated with the first and 
second mode processes modalities . 

23. (Currently Amended) The method of claim 20, wherein the step of triggering 
comprises the steps of: 

looking up the received command in the registration table; and 
executing the actions associated with the received command m by the first and second 
mode processes modalities . 
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24. (Currently Amended) The method of claim 20, further comprising the steps of: 
registering a callback handle for each of the registered commands to notify the first and 

second mode processes modalities of completion of the actions corresponding to the registered 
commands. 

25. (Currently Amended) The method of claim 24, wherein the step of updating the 
application states or the device states comprises the steps of executing the callback handle 
associated with the received command to trigger a callback action on the first mode process 
modality and a callback action on the second mode process modality , 

26. (Currently Amended) The method of claim 19, wherein the step of triggering 
comprises the steps of: 

executing first thread associated with the received command; and 
triggering a corresponding second thread to initiate the corresponding action en by the at 
least second mode process modality . 

27. (Original) The method of claim 26, wherein the threads are applets. 

28. (Original) The method of claim 26, wherein the threads communicate via socket 
connections. 

29. (Currently Amended) A program storage device readable by a machine, tangibly 
embodying a program of instructions executable by the machine to perform method steps for 
synchronizing multi- modal interactions, the method comprising the steps of: 

activating a multi-modal application comprising at least a first mode process that enables 
user interaction with the application in a first modality and a second mode process that enables 
user interaction with the application in a second modality 

receiving a command or event in the first modality; 

triggering (i) an action in by the first modality mode process based on the received 
command or event and (ii) triggering a corresponding action m by the second mode process 
modality, based on th e received command or oyont ; and 
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updating application states or device states associated with the first and second mode 
processes modality and th e s e cond modality . 

30. (Currently Amended) The program storage device of claim 29, further comprising 
instructions for performing the steps of: 

registering active commands associated with the first mode process modality and active 
commands associated with the second mode process modality ; 

associating, with each registered command of the first mode process modality , an action 
on the first mode process modality and a corresponding action on the second mode process 
modality ; and 

associating, with each registered command of the second mode process modality , an 
action on the second mode process modality and a corresponding action on the first mode 
process modality . 

31. (Original) The program storage device of claim 30, further comprising instructions 
for performing the step of building a command-to-action registration table based on the 
registered commands and actions. 

32. (Currently Amended) The program storage device of claim 30, wherein the 
instructions for performing the step of triggering comprise instructions for performing the steps 

of: 

looking up the received command in the registration table; and 
executing the actions associated with the received command en by the first and second 
mode processes modaliti e s . 

33. (Currently Amended) The program storage device of claim 30, further comprising 
instructions for performing the steps of: 

registering a callback handle for each of the registered commands to notify the first and 
second mode processes modalities of completion of the actions corresponding to the registered 
commands. 
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34. (Currently Amended) The program storage device of claim 33, wherein the 
instructions for performing the step of updating the application states or the device states 
comprise instructions for performing the step of executing the callback handle associated with 
the received command to trigger a callback action on the first mode process modality and a 
callback action on the second mode process modality . 

35. (Currently Amended) The program storage device of claim 29, wherein the 
instructions for performing the step of triggering comprise instructions for performing the steps 

of: 

executing first thread associated with the received command; and 
triggering a corresponding second thread to initiate the corresponding action en by the at 
least second mode process modality . 

36. (Original) The program storage device of claim 35, wherein the threads are applets. 

37. (Original) The program storage device of claim 35, wherein the threads communicate 
via socket connections. 
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